1 Introduction to stochastic processes

2 Time series

Observations:

• \(y_t \in \mathbb{R}\hspace{0.08cm}\) is a realization of the random variable \(\\hspace{0.08cm}mathcal{Y}_t\) \(\\[0.35cm]\)

• A time series is a realization of a stochastic process. The time series is considered a result or trajectory of the stochastic process. \(\\[0.35cm]\)

• A time series can be defined as a vector of data points ordered in time. Where the data is equally spaced in time, namely, between each data point there is the same time space, such as a week, a month, a trimester, a quarter …

The process is characterized by the join probability distribution of the random variables \(\hspace{0.1cm} \mathcal{Y}_1 , \mathcal{Y}_2 ,..., \mathcal{Y}_k \hspace{0.1cm}\), namely, is characterized by the join density or probability function \(\hspace{0.08cm}f_{\hspace{0.08cm}\mathcal{Y}_1 , \mathcal{Y}_2 ,..., \mathcal{Y}_k}\).

This distribution is called finite-dimensional distribution of the process. We say that we know the probabilistic structure of the stochastic process when we know that join distribution, which determine the distribution of any subset of the variables and, in particular, the marginal distribution of each variable.

3 Mean function

Given a stochastic process \(\hspace{0.1cm} \Bigl\{ \hspace{0.05cm} \mathcal{X}_t \hspace{0.1cm}: \hspace{0.1cm} t \in T=\lbrace 1,2,...,k \rbrace \hspace{0.05cm} \Bigr\} \hspace{0.1cm}\)

The mean function $ _t $ of the process is defined as:

\[\mu_t = E\left[\mathcal{X}_t\right]\]

for \(\hspace{0.1cm} t \in \lbrace 1,2,...,k \rbrace \\\)

Observations:

• An important particular case, due to its simplicity, arises when all the variables have the same mean and thus the mean function is a constant. The realizations of the process show no trend and we say that the process is stable in the mean. \(\\[0.35cm]\)

• If, on the contrary, the means change over time, the observations at dierent moments will reveal that change. \(\\[0.35cm]\)

• On many occasions we only have one realization of the stochastic process and we have to deduce from that whether the mean function of the process is, or is not, constant over time.

4 Variance function

Given a stochastic process \(\hspace{0.1cm}\left\lbrace \hspace{0.1cm} \mathcal{X}_t \hspace{0.1cm}/\hspace{0.1cm} t \in T=\lbrace 1,2,...,k \rbrace \hspace{0.1cm}\right\rbrace \hspace{0.1cm}\)

The variance function \(\sigma^2_t\) of the process is defined as:

\[\sigma^2_t = Var\left[\mathcal{X}_t\right]\]

for \(t \in \lbrace 1,2,...,k \rbrace \\\)

We say that the process is stable in the variance if the variability is constant over time.

A process can be stable in the mean but not in the variance and vice versa.

5 Autocovariance function

The structure of linear dependence between random variables is represented by the covariance and correlation functions.

Given a stochastic process \(\hspace{0.1cm}\left\lbrace \hspace{0.1cm} \mathcal{X}_t \hspace{0.1cm}/\hspace{0.1cm} t \in T=\lbrace 1,2,...,k \rbrace \hspace{0.1cm}\right\rbrace \hspace{0.1cm}\)

The autocavariance function \(\gamma_{t , t+h}\) of the process is defined as:

\[\gamma_{t , t+h} = Cov(\mathcal{X}_t , \mathcal{X}_{t+h}) = E\left[ (\mathcal{X}_t - \mu_t)\cdot (\mathcal{X}_{t+h} - \mu_{t+h}) \right]\]

for \(\hspace{0.1cm}t \in \lbrace 1,2,...,k \rbrace\hspace{0.1cm}\) and \(\hspace{0.1cm} h\in \lbrace 1,2,... \rbrace \\\)

In particular, we have

\[\gamma_{t , t} = \sigma_t^2\]

The autocovariances have dimensions, the squares of the series, thus it is not advisable to use them for comparing series measured in dierent units.

6 Autocorrelation function

Given a stochastic process \(\hspace{0.1cm}\left\lbrace \hspace{0.1cm} \mathcal{X}_t \hspace{0.1cm}/\hspace{0.1cm} t \in T=\lbrace 1,2,...,k \rbrace \hspace{0.1cm}\right\rbrace \hspace{0.1cm}\)

The autocorrelation function \(\rho_{t , t+h}\) of the process is defined as:

\[\rho_{t , t+h} = \dfrac{\gamma_{t , t+h}}{\sqrt{\sigma_t^2 \cdot \sigma_{t+h}^2}}\]

for \(\hspace{0.1cm}t \in \lbrace 1,2,...,k \rbrace\hspace{0.1cm}\) and \(\hspace{0.1cm} h\in \lbrace 1,2,... \rbrace \\\)

In particular, we have

\[\rho_{t , t} = 1\]

It is interesting to notice the differences between conditional distributions and the marginal distributions.

The marginal distribution of \(\mathcal{X}_t\) represents what we know about a variable, without knowing anything about its trajectory until time \(t\).

The conditional distribution of \(\mathcal{X}_t\) given \(\mathcal{X}_{t-1}\),…,\(\mathcal{X}_{t-r}\) represents what we know about a variable when we know the k previous values of the process.

In time series conditional distributions are of greater interest than marginal ones because they define the predictions that we can make about the future knowing the past.

7 Visualization of time series in Python

Throughout this article we will use a time series on sales of a company.

First of all, we load some of the libraries that we are to use:

import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
import seaborn as sns

sns.set_theme()

sns.set(rc={'figure.figsize':(20,9)})

We load the data-set:

Time_Series_1 = pd.read_csv('Time_Series_1.csv')

Time_Series_1

The default periodicity of this time series is daily.

We can group a time series by different periods, as hours, days, weeks, months, quarter, years…

Concretely we will group this time series by day, week, month and quarter.

We can get date column as follows:

Time_Series_1['Fecha'] = Time_Series_1['CODTAVEN'].str[0:10]

We have to convert date column to date format:

Time_Series_1['Fecha'] = pd.to_datetime(Time_Series_1['Fecha'])

We can create the columns Day, Week, Month, Quarter and Year as follows:

Time_Series_1['Dia'] = Time_Series_1['Fecha'].dt.day

Time_Series_1['Semana'] = Time_Series_1['Fecha'].dt.week

Time_Series_1['Mes'] = Time_Series_1['Fecha'].dt.month

Time_Series_1['Trimestre'] = Time_Series_1['Fecha'].dt.quarter

Time_Series_1['Año'] = Time_Series_1['Fecha'].dt.year

We select the columns with which we are going to work. IMPVENTA will be the response variable, namely, the variable we want to predict.

Time_Series_1 = Time_Series_1.loc[: , ['Fecha', 'Dia', 'Semana', 'Mes', 'Trimestre', 'Año', 'IMPVENTA']]

Time_Series_1

Monthly_Time_Series_1 = Time_Series_1.groupby(['Año', 'Mes'])['IMPVENTA'].sum().reset_index(drop=False)

Monthly_Time_Series_1

Monthly_Time_Series_1['Año'] = Monthly_Time_Series_1['Año'].astype('string')
Monthly_Time_Series_1['Mes'] = Monthly_Time_Series_1['Mes'].astype('string')

Monthly_Time_Series_1['Mes-Año'] = Monthly_Time_Series_1[['Mes', 'Año']].agg('-'.join, axis=1)

Monthly_Time_Series_1

fig, ax = plt.subplots()

p=sns.lineplot(x="Mes-Año", y="IMPVENTA", data=Monthly_Time_Series_1 , color='red')

plt.setp(p.get_xticklabels(), rotation=90)

plt.title("Monthly Time Series",  fontsize = 17)

fig.savefig('p1.jpg', format='jpg', dpi=1200)

plt.show()

Daily_Time_Series_1 = Time_Series_1.groupby(['Año', 'Mes','Dia'])['IMPVENTA'].sum().reset_index(drop=False)

Daily_Time_Series_1

Daily_Time_Series_1['Año'] = Daily_Time_Series_1['Año'].astype('string')
Daily_Time_Series_1['Mes'] = Daily_Time_Series_1['Mes'].astype('string')
Daily_Time_Series_1['Dia'] = Daily_Time_Series_1['Dia'].astype('string')

Daily_Time_Series_1['Dia-Mes-Año'] = Daily_Time_Series_1[['Dia', 'Mes', 'Año']].agg('-'.join, axis=1)

Daily_Time_Series_1

fig, ax = plt.subplots()

p=sns.lineplot(x="Dia-Mes-Año", y="IMPVENTA", data=Daily_Time_Series_1 , color='red')

p.set_xticks(np.arange(0 , len(Daily_Time_Series_1) , 40))

plt.setp(p.get_xticklabels(), rotation=90)

plt.title("Daily Time Series", fontsize = 20)

fig.savefig('p2.jpg', format='jpg', dpi=1200)

plt.show()

Weekly_Time_Series_1 = Time_Series_1.groupby(['Año', 'Mes','Semana'])['IMPVENTA'].sum().reset_index(drop=False)

Weekly_Time_Series_1['Año'] = Weekly_Time_Series_1['Año'].astype('string')
Weekly_Time_Series_1['Mes'] = Weekly_Time_Series_1['Mes'].astype('string')
Weekly_Time_Series_1['Semana'] = Weekly_Time_Series_1['Semana'].astype('string')

Weekly_Time_Series_1['Semana-Mes-Año'] = Weekly_Time_Series_1[['Semana', 'Mes', 'Año']].agg('-'.join, axis=1)

fig, ax = plt.subplots()

p=sns.lineplot(x="Semana-Mes-Año", y="IMPVENTA", data=Weekly_Time_Series_1 , color='red')

p.set_xticks(np.arange(0 , len(Weekly_Time_Series_1) , 5))

plt.setp(p.get_xticklabels(), rotation=90)

plt.title("Weekly Time Series",  fontsize = 17)

fig.savefig('p3.jpg', format='jpg', dpi=1200)

plt.show()

Quarter_Time_Series_1 = Time_Series_1.groupby(['Año', 'Trimestre'])['IMPVENTA'].sum().reset_index(drop=False)

Quarter_Time_Series_1

Quarter_Time_Series_1['Año'] = Quarter_Time_Series_1['Año'].astype('string')
Quarter_Time_Series_1['Trimestre'] = Quarter_Time_Series_1['Trimestre'].astype('string')

Quarter_Time_Series_1['Trimestre-Año'] = Quarter_Time_Series_1[['Trimestre', 'Año']].agg('-'.join, axis=1)

fig, ax = plt.subplots()

p=sns.lineplot(x="Trimestre-Año", y="IMPVENTA", data=Quarter_Time_Series_1 , color='red')

p.set_xticks(np.arange(0 , len(Quarter_Time_Series_1) , 1))

plt.setp(p.get_xticklabels(), rotation=90)

plt.title("Quarter Time Series",  fontsize = 17)

fig.savefig('p4.jpg', format='jpg', dpi=1200)

plt.show()

8 Time Series Descomposition

Time series decomposition is a process by which we separate a time series into its components: trend, seasonality and residuals.

• Trend represents the slow-moving changes in a time series. \(\\[0.25cm]\)

• Seasonality represent the seasonal pattern in the series. The cycles occur repeatedly over a fixed period of time. \(\\[0.25cm]\)

• Residuals represent the behavior that cannot be explained by the trend and seasonality components. \(\\[0.25cm]\)

9 Time Series Forecasting

\(\hspace{0.25cm}\) Given a stochastic process \(\hspace{0.15cm} \mathcal{Y} \hspace{0.1cm}=\hspace{0.1cm} \Bigl( \hspace{0.06cm} \mathcal{Y}_t \hspace{0.12cm}: \hspace{0.12cm} t \in T=\lbrace 1,2,...,n \rbrace \hspace{0.06cm}\Bigl) \hspace{0.1cm} = \hspace{0.1cm}\Bigl( \hspace{0.06cm} \mathcal{Y}_1 , \mathcal{Y}_2 ,..., \mathcal{Y}_n \hspace{0.06cm}\Bigl) \hspace{0.1cm}\)

\(\hspace{0.25cm}\) We have a time series \(\hspace{0.07cm}Y_t = \left( y_1, y_2, ...,y_n \right)^t \hspace{0.1cm}\) associated to the process \(\hspace{0.07cm}\mathcal{Y}\hspace{0.07cm}\).

So, \(\hspace{0.07cm}y_t\hspace{0.07cm}\) is the value of the process \(\hspace{0.07cm}\mathcal{Y}\hspace{0.07cm}\) in the period \(\hspace{0.07cm}t\).

Our goal is to predict the future of the process. For that we will use the information on the process given by the time series, namely, using the available data.

We want to predict \(\hspace{0.07cm}y_{n+k}\hspace{0.07cm}\), for \(\hspace{0.07cm}k=1,2,3,\dots\).

The idea under most part of forecasting methods is to predict \(\hspace{0.07cm}y_{n+k}\hspace{0.07cm}\) as:

\(\widehat{y}_{n+k} \hspace{0.07cm}=\hspace{0.07cm} \widehat{g}(y_1, y_2, ...,y_n)\)

Observations:

• We are assuming that unique available data is \(\hspace{0.07cm}Y_t = \left( y_1, y_2, ...,y_n \right)^t \hspace{0.1cm}\) \(\\[0.2cm]\)

• If \(\hspace{0.07cm}y_n\hspace{0.07cm}\) is the value of the process \(\hspace{0.07cm}\mathcal{Y}\hspace{0.07cm}\) corresponding to most recent period, \(\hspace{0.07cm}y_{n+k}\hspace{0.07cm}\) is the value of the process in a future period. So, by definition, we don’t know \(\hspace{0.07cm}y_{n+k}\hspace{0.07cm}\), for \(\hspace{0.07cm}k=1,2,3,\dots\).

10 Regression Problem vs Time Series Forecasting

You probably have encountered regression problems where you must predict some continuous target given a certain set of features. At first glance, time series forecasting seems like a typical regression problem: we have some historical data, and we wish to build a mathematical expression that will express future values as a function of past values. However, there are some key differences between time series forecasting and regression for time-independent scenarios that deserve to be addressed before we look at our very first forecasting technique.

• Time series have an order

  The first concept to keep in mind is that time series have an order, and we cannot change that order when modeling. In time series forecasting, we express future values as a function of past values. Therefore, we must keep the data in order, so as to not violate this relationship.

  Other regression tasks in machine learning often do not have an order. For example, if you are tasked to predict revenue based on ad spend, it does not matter when a certain amount was spent on ads. Instead, you simply want to relate the amount of ad spend to the revenue. In fact, you might even randomly shuffle the data to make your model more robust. Here the regression task is to simply derive a function such that given an amount on ad spend, an estimate of revenue is returned. On the other hand, time series are indexed by time, and that order must be kept.

• Time series sometimes do not have features

  It is possible to forecast time series without the use of features other than the time series itself. As data scientists, we are used to having datasets with many columns, each representing a potential predictor for our target. For example, consider the task of predicting revenue based on ad spend, where the revenue is the target variable. As features, we could have the amount spent on Google ads, Facebook ads, and television ads. Using these three features, we would build a regression model to estimate revenue. However, with time series, it is quite common to be given a simple dataset with a time column and a value at that point in time. Without any other features, we must learn ways of using past values of the time series to forecast future values. This is when the moving average model or autoregressive model come into play, as they are ways to express future values as a function of past values

11 Baseline Models

12 Stationary processes

ver tbn lo de Maria dolores

Given a stochastic process \(\hspace{0.1cm}\left\lbrace \hspace{0.1cm} \mathcal{X}_t \hspace{0.1cm}/\hspace{0.1cm} t \in T=\lbrace 1,2,...,k \rbrace \hspace{0.1cm}\right\rbrace \hspace{0.1cm}\)

A stochastic process is strictly stationary if:

the probability distribution of \(\mathcal{X}_{t}\) is the same as that of \(\mathcal{X}_{t+h}\)

for all \(\hspace{0.1cm}t \in \lbrace 1,2,...,k \rbrace\hspace{0.1cm}\) and \(\hspace{0.1cm} h \in \in \lbrace 1,2,... \rbrace\).

Therefore, for all set of times \(\hspace{0.1cm}t_1 , t_2,...,t_n\)

\((\mathcal{X}_{t_1}, \mathcal{X}_{t_2},\dots ,\mathcal{X}_{t_n} )\hspace{0.1cm}\) is identically distributed as \(\hspace{0.1cm}(\mathcal{X}_{t_1+h}, \mathcal{X}_{t_2+h},\dots ,\mathcal{X}_{t_n+h} )\)

Strict stationarity is a very strong condition, since to prove it we must have the joint distributions for any set of variables in the process. A weaker property, but one which is easier to prove, is weak stationarity.

A stochastic process is weakly stationary if:

• \(\mu_t = \mu = cte , \forall t \in \lbrace 1,2,...,k \rbrace\)

• \(\sigma_t^2 = \sigma = cte , , \forall t \in \lbrace 1,2,...,k \rbrace\)

• \(\gamma_{t , t + h} = Cov(\mathcal{X}_t,\mathcal{X}_{t+h}) E[(\mathcal{X}_t - \mu)\cdot (\mathcal{X}_{t+h} - \mu)] = \gamma(h) , \forall h \in \lbrace 0 , \pm 1 , \pm 2 ,... \rbrace\)

The first two conditions indicate that the mean and variance are constant.

The third indicates that the covariance between two variables depends only on their separation.

In a stationary process the autocovariances and autocorrelations depend only on the lag between the variables and, in particular, the relationship between \(\mathcal{X}_t\) and \(\mathcal{X}_{t+h}\) , is always equal to the relationship between \(\mathcal{X}_t\) and \(\mathcal{X}_{t-h}\) .

As a result, in stationary processes:

\(\gamma_{t , t + h} = \gamma_{t + r , t + h + r} = \gamma(h) , \forall r \in \lbrace 0 , \pm 1 , \pm 2 ,... \rbrace\)

\[\rho_{t, t+h} = \dfrac{\gamma_{t , t + h}}{\sqrt{\sigma_t^2 \cdot \sigma_{t+h}^2}} = \dfrac{\gamma(h)}{\sqrt{\sigma^2 \cdot \sigma^2}} = \dfrac{\gamma(h)}{\sigma^2} = \dfrac{\gamma(h)}{\gamma(0)} = \rho(h)\]

Where:

\(\gamma(0) = \sigma^2\)

13 White Noise Process

14 Transformation to get stationarity

ver tbn lo de Maria dolores

15 Testing for stationarity

ver tbn lo de Maria dolores

Dickey-fuller test

Aumented Dickey Fuller test

16 Random walk

17 Moving Average Process: MA

Para un modelo MA(1), la función de verosimilitud toma la forma:

\[\mathcal{L}(\theta_1, \sigma^2) = \prod_{t=1}^T \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_t - \theta_1 \epsilon_{t-1})^2}{2\sigma^2}\right)\]

donde \(\epsilon_t\) es un término de error gaussiano con media cero y varianza \(\sigma^2\), y \(\theta_1\) es un parámetro del modelo. Nuevamente, la función de log-verosimilitud es:

\[\ell(\theta_1, \sigma^2) = -\frac{T}{2} \ln(2\pi) - \frac{T}{2} \ln(\sigma^2) - \frac{1}{2\sigma^2} \sum_{t=1}^T (y_t - \theta_1 \epsilon_{t-1})^2\]

El objetivo es encontrar los valores de los parámetros \(\theta_1\) y \(\sigma^2\) que maximizan la función de log-verosimilitud. Al igual que en los casos anteriores, se puede utilizar el método de Máxima Verosimilitud para estimar los parámetros, encontrando los valores de \(\theta_1\) y \(\sigma^2\) que maximizan \(\ell(\theta_1, \sigma^2)\). Esto también se puede lograr mediante métodos numéricos como el método de Newton-Raphson o el método de gradiente descendente.

Para un modelo MA(q), la función de verosimilitud toma la forma:

\[\mathcal{L}(\theta_1, \ldots, \theta_q, \sigma^2) = \prod_{t=q+1}^T \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_t - \sum_{i=1}^q \theta_i \epsilon_{t-i})^2}{2\sigma^2}\right)\]

donde \(\epsilon_t\) es un término de error gaussiano con media cero y varianza \(\sigma^2\), y \(\theta_1, \ldots, \theta_q\) son parámetros del modelo. Nuevamente, la función de log-verosimilitud es:

\[\ell(\theta_1, \ldots, \theta_q, \sigma^2) = -\frac{T-q}{2} \ln(2\pi) - \frac{T-q}{2} \ln(\sigma^2) - \frac{1}{2\sigma^2} \sum_{t=q+1}^T (y_t - \sum_{i=1}^q \theta_i \epsilon_{t-i})^2\]

El objetivo es encontrar los valores de los parámetros \(\theta_1, \ldots, \theta_q\) y \(\sigma^2\) que maximizan la función de log-verosimilitud. Al igual que en los casos anteriores, se puede utilizar el método de Máxima Verosimilitud para estimar los parámetros, encontrando los valores de \(\theta_1, \ldots, \theta_q\) y \(\sigma^2\) que maximizan \(\ell(\theta_1, \ldots, \theta_q, \sigma^2)\). Esto también se puede lograr mediante métodos numéricos como el método de Newton-Raphson o el método de gradiente descendente.

18 Autorregresive Process: AR

Para estimar los parámetros de un modelo AR(1), se puede utilizar el método de Máxima Verosimilitud. Dado un conjunto de datos \(y_t\) para \(t=1,\ldots,T\) y un modelo AR(1) dado por:

\[ y_t = \phi_1 y_{t-1} + \epsilon_t \]

donde \(\epsilon_t \sim N(0,\sigma^2)\) es un término de error gaussiano con media cero y varianza \(\sigma^2\), se busca encontrar los valores de los parámetros \(\phi_1\) y \(\sigma^2\) que maximizan la función de verosimilitud:

\[ \mathcal{L}(\phi_1, \sigma^2) = \prod_{t=2}^T \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_t - \phi_1 y_{t-1})^2}{2\sigma^2}\right) \]

La función de log-verosimilitud es entonces:

\[ \ell(\phi_1, \sigma^2) = -\frac{T-1}{2} \ln(2\pi) - \frac{T-1}{2} \ln(\sigma^2) - \frac{1}{2\sigma^2} \sum_{t=2}^T (y_t - \phi_1 y_{t-1})^2 \]

La estimación de los parámetros se realiza maximizando la función de log-verosimilitud, es decir, encontrando los valores de \(\phi_1\) y \(\sigma^2\) que hacen que \(\ell(\phi_1, \sigma^2)\) sea lo más grande posible. Esto se puede hacer de varias maneras, por ejemplo, mediante métodos numéricos como el método de Newton-Raphson o el método de gradiente descendente.

¡Por supuesto! Para un modelo AR(p), la función de verosimilitud toma la forma:

\[\mathcal{L}(\boldsymbol{\phi}, \sigma^2) = \prod_{t=p+1}^T \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y_t - \sum_{i=1}^p \phi_i y_{t-i})^2}{2\sigma^2}\right)\]

donde \(\boldsymbol{\phi} = (\phi_1, \ldots, \phi_p)\) es un vector de parámetros, y \(y_t\) es una serie de tiempo para \(t=1, \ldots, T\). De nuevo, la función de log-verosimilitud es:

\[ \ell(\boldsymbol{\phi}, \sigma^2) = -\frac{T-p}{2} \ln(2\pi) - \frac{T-p}{2} \ln(\sigma^2) - \frac{1}{2\sigma^2} \sum_{t=p+1}^T (y_t - \sum_{i=1}^p \phi_i y_{t-i})^2 \]

El objetivo es encontrar los valores de los parámetros \(\boldsymbol{\phi}\) y \(\sigma^2\) que maximizan la función de log-verosimilitud. Al igual que en el caso AR(1), se puede utilizar el método de Máxima Verosimilitud para estimar los parámetros, encontrando los valores de \(\boldsymbol{\phi}\) y \(\sigma^2\) que maximizan \(\ell(\boldsymbol{\phi}, \sigma^2)\). Esto también se puede lograr mediante métodos numéricos como el método de Newton-Raphson o el método de gradiente descendente.